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General Series Introduction 

Orsett Technical Reports are designed to allow the 
exploration of specific topics in detail. Series A 
contains four reports on different aspects of the student 
evaluation of teaching effectiveness (SETE) or students' 
ratings of instruction (SRI) . This is the rating of 
lecturers and teachers by their students. 

REPORT No.l ' 

This report is a literature review of the studies 
into SETE and SRI, mostly from the USA. The aim is to 
outline what students see as the "ideal lecturer". Much 
of the material comes from the prolific work of Kenneth 
Feldman . 

REPORT No. 2 2 

This report addresses the issue of the accuracy of 
students' ratings of their instructors. Is it an accurate 
picture of their teaching effectiveness or the personal 
feelings of the students? The issues of reliability, 
generalisability , and validity of the ratings, along with 
rating errors, are discussed. 

REPORT No. 3 3 

Report no . 3 takes many of the technical issues 
raised in report no . 2 further. In particular, the 
potential biases to SETE and SRI . 

REPORT No. 4 4 

This report gives details of the construction of the 
Birmingham Overseas Student Teaching Evaluation 
Questionnaire (BOSTEQ) . The aim is to produce a rating 
instrument specifically to be used by overseas students . 

The research is part of an MSc degree at the 
University of Aston 5 . 



1 Brewer, K (2002a) Student evaluation of teaching effectiveness: an introduction, Orsett Technical 
Reports, Series A, No.l, Orsett Psychological Services: Orsett, Essex. 

Brewer, K (2002b) Student evaluation of teaching effectiveness: methodological issues - part 1, 
Orsett Technical Reports, Series A, No. 2, Orsett Psychological Services: Orsett, Essex. 

Brewer, K (2002c) Methodological issues with student evaluation of teaching effectiveness (SETE) 
- part 2, Orsett Technical Reports, Series A, No. 3, Orsett Psychological Services: Orsett, Essex. 
4 Brewer, K (2002d) Construction of Birmingham Overseas Students Teaching Evaluation 
Questionnaire (BOSTEQ), Orsett Technical Reports, Series A, No. 4, Orsett Psychological Services: 
Orsett, Essex. 

Brewer, K (1993) Overseas Students Evaluation of Teaching Effectiveness, Unpublished MSc 
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INTRODUCTION 

The major studies in the literature of Student 
Evaluation of Teaching Effectiveness (SETE) have used 
factor analysis to reduce the number of characteristics 
of the ideal lecturer. For example, Smithers (1970a) 
starting with a list of fifty characteristics of the 
ideal lecturer was able to reduce them to twelve rotated 
factors . 

Among the rating instruments, the Student 
Educational Evaluation Questionnaire (SEEQ) proposed by 
Marsh, has 9 factors: "learning/value"; "instructor 
enthusiasm"; "organisation"; "individual rapport"; "group 
interaction"; "breadth of coverage"; "exams/grading"; 
"assignment /reading" ; and "workload/difficulty". These 
factors have most recently been confirmed by Marsh and 
Hocevar (1991) . 

Feldman (1976b) summarising the research on the 
characteristics of the ideal lecturer reduces 19 
characteristics to three groupings: 

i) Presentation - stimulation; enthusiasm; 
knowledge; preparation; clarity; overall evaluation; and 
course materials; intellectual expansiveness; elocution; 
sensitivity . 

ii) Facilitation - respect; encouragement of 
questions; intellectual challenge; availability. 

iii) Regulation - not so clear. 

METHODOLOGY 



From the literature surveyed (see Brewer 2002a), 
over 100 different characteristics of the ideal lecturer 
were amassed. Similar items with slightly different 
wording were combined - for example, "explains criticism 
of student performance" and "makes helpful and 
constructive comments on essays"; "encourages to 
thinking" and "stimulates independent thought". 

Items relating directly to course, reading or 
tests/assignments were removed. This left items that 
related directly to the lecturer, and the lecture 
situation: a total of 62 items (which are listed in full 
in Appendix 1 ) . 



Construction of Birmingham Overseas Students Teaching Evaluation Questionnaire (BOSTEQ) 
ISBN: 978-0-9540761-7-7 Kevin Brewer 2002 



Pilot Study 

Because none of the past research has looked solely 
at overseas students, it was uppermost not to assume what 
characteristics they would choose as important for the 
ideal lecturer. 

Thus the first section of the pilot study was a 
simple open question about characteristics of the ideal 
lecturer. It was used to check if any characteristics 
important to the students had not been included in the 
prescribed items. Nothing new was volunteered. 

The main section involved the 62 items. The task was 
to rank in order of importance, the top 20. It was 
accepted that this was a difficult task. Ranking seemed 
better than rating every item on a scale of 1 to 6. The 
variation of rating was tried with a small number of 
questionnaires, but as expected, it was hard to 
distinguish between the items. 

Furthermore, because of the fear of social 
desirability, which it was felt could be a major problem 
with some nationalities, ratings were avoided. 

Experience has shown that in some parts of the 
world, making any comment that could be construed as a 
criticism of the teacher, is unacceptable. In many 
cultures, particularly in Asia, the teacher has a very 
high status (eg: Regmi 1987) . This also poses a problem 
for the final rating questionnaire - whether the ratings 
of a lecturer are a true comment by the student or a 
socially desirable answer? Certainly further research 
would need to try to isolate if there is an effect of 
this potential bias. 

It was suggested that the 62 items should have been 
grouped together in certain ways, in order to ease the 
ranking. But because of the desire not to influence the 
results in any way, the 62 items were randomly listed. 
The sample used for the pilot study were all approached 
individually, and known to the researcher; so it was 
hoped this would encourage them to spend the extra effort 
on ranking the items. They were all students at the 
University of Birmingham or Aston Business School, and 
twenty-four questionnaires were returned. 

The nationality make-up of returned questionnaires 
of the pilot study were as follows: 11 Japanese; 3 
Brazilian; 2 Indonesian; 2 Taiwanese; 1 Dutch; 1 German; 
2 Greek; 1 Portuguese; and 1 Thai. This included 16 males 
and 8 females . 

The sample came from the following departments: 
English for Overseas Students Unit (11); International 
Studies (2); Development Finance (2); Aston Business 
School (5); Mechanical Engineering (1); Psychology ( 1 ) ; 
Economics (1) ; and Law (1) . 
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From their responses the five most popular items 
were: "well-organised and clearly presented lectures"; 
"has good knowledge of own subject"; "is well-prepared 
for classes"; "structures difficult topics in easily 
understandable ways"; and "has good general knowledge". 

With the least important items being: "makes good 
use of the overhead projector"; "always well-groomed, 
neat and clean in personal appearance"; "skilful in 
blackboard drawing"; "keeps strictly to textbook in 
lectures"; "has good vocabulary"; and "includes many 
relevant personal anecdotes". 



Final study 

The majority of the approximately 2000 overseas 
students in Birmingham, were studying at the University 
of Birmingham. 

Based on university statistics for the academic year 
1992-3, it is estimated that the largest nationality 
grouping (39.2%) of the total overseas students at the 
University of Birmingham come from South-east Asia. The 
largest departmental grouping is Accounting and Business 
(18.7% of total overseas students) . 

The final questionnaire was made up of the top 
twenty items chosen from the pilot study, plus two of the 
least popular items as controls. The total was 22 items 
to rank (see copy in Appendix 2) . 

969 questionnaires were sent out, through the 
internal post at the University of Birmingham to the 
departments of the sample. The response was 165 (response 
rate of 17%) . But not all the responses were complete, 
some had information missing. Thus the figures for 
frequencies vary slightly. 

The response was felt to be disappointing. However, 
it was not possible to send a follow-up letter because it 
was not clear whether the list of overseas students being 
used was up-to-date. The list was compiled in October 
1992, but not used until the following March. Thus a 
number of students may have left the University. 



RESULTS 

The aim was for the original sample to mirror the 
make-up of overseas students at the University of 
Birmingham. This was so, except for Europe. A large 
number of the students were short term for 3 months, and 
had returned home by the time of the questionnaire 
(March) . Otherwise the returns mirror the make-up 
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of the overseas student population fairly closely. This 
can be seen in table 1. For example, students from 
South-east Asia make up 39.2% of all overseas students at 
the University of Birmingham, and 43% of the returned 
questionnaires were from this group. 

Though the overall response rate was below 20%, 
which was felt to be a little disappointing. It was 
accepted that this can be the case for many postal 
questionnaires (Bridge 1974; Monette et al 1990) . The 
responses do give a good cross-section of the overseas 
students in the University of Birmingham. 



NATIONAL NUMBER % OF TOTAL NATIONAL NATIONAL 

GROUPING OF RETURN GROUPING GROUPING 

RETURNS AS % OF AS % OF ALL 

SAMPLE OVERSEAS 













STUDENTS 












AT B.U 


South-East Asia 


71 


43.0 


40.8 




39.2 


Indian Sub-continent 


12 


7.2 


9.8 




6.1 


Africa 


36 


21.8 


21.9 




17.4 


Arabic- speaking 












countries 


8 


4.8 


7.4 




4.4 


Europe 


17 


10.3 


10.3 




25.3 


Caribbean/N America 


7 


4.2 


7.6 




4.5 


C and S America 


9 


5.4 


2.2 




1.8 


Missing details 


5 


3.3 


- 




- 


TOTAL 


165 


100 


100 . 


;969) 


100 (164 



Table 1 - showing the returns of questionnaire by 
nationality . 



The participants were asked to rank the 22 
characteristics in terms of importance for the "ideal 
lecturer" (1 = most important) . Then the results were 
scored as 22 points for 1 through to 1 point for the 22nd 
item. The items chosen for inclusion were the top 20 
ranked from the pilot study, plus the two least popular 
as controls . 

It could be argued that the data is ordinal. But it 
is necessary to class it as interval to allow the use of 
parametric tests. Thus we assume the difference between 
each score is equal distance, and there is no absolute 
zero (Stone and James 1965; see also Coolican 1990; Kenny 
1986; Miller 1983; Miller 1984; Monette et al 1990; 
Runyon and Haber 1977) . The analysis was by SPSS PC 
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(version 4.0.1 SPSS Inc 1989). 

Table 2 shows the final ranking of the 
characteristics based on mean score. (See Cranton and 
Smith 1990 for reasoning behind choosing mean score) . The 
top five characteristics chosen were "well-organised and 
clearly presented lectures"; "is well-prepared for 
classes"; "explains concepts, principles and abstract 
theories clearly and precisely"; "has good knowledge of 
subject"; and "structures difficult topics in easily 
understandable ways". Not surprisingly "speaks clearly 
and audibly" is also important. Items in 1st and 2nd 
position are quite similar, as are those in 3rd and 5th 
position . 



Characteristics 



Mean 



1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

11 

13 

14 

15 

16 

17 

18 

19 

20 

21 

22 



Organised (G) . 18.24 

Prepared (D) . 16.74 

Explains concepts (H) . 16.47 

Knowledge of subject (F) . 15.83 

Structures difficult topics (E) . 15.55 

Speaks clearly (P). 13.54 

Objectives (L) . 13.00 

Aware (Q) . 12 .24 

Stimulates independent thought (J) . 12.29 

. Handouts (A) . 12.22 

. = Express themselves (M) . 11.73 

. = Feedback (N) . 11.73 

. Developments (B) . 11.35 

. Motivates students (R) . 10.77 

. Welcomes guestions (U) . 10.39 

. Interest in subject (S) . 10.29 

. Relate to other subjects (I) . 10.25 

. Answers guestions accurately (K) . 10.22 

. General knowledge (C). 10.04 

. Sense of humour (T) . 7.13 

. Good vocabulary (V) . 4 . 92 

. Personal anecdotes (0) . 4.65 



Table 2 - showing most popular items of ideal lecturer in 
rank order. The full characteristics are listed in 
Appendix 1. The letters in brackets refer to the position 
on the final questionnaire used. 



Thus the most important characteristics of lecturers 
for overseas students can be summarised as: 

i) Prepared and organised lectures, 
ii) Clear explanations of topics. 

This compares closely with the earlier British 
studies, but slightly different to the North American 
studies. Feldman (1976b) found "respect for students"; 
"clarity of presentation"; "availability outside class"; 
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"knowledge of subject"; "concern for class progress"; and 
"interesting to students" to be the most important items, 
depending on the type of study. 

It was hoped at the start of the research that there 
would be nationality differences in the ranking of 
characteristics. Unfortunately the number of responses 
from certain nationality groupings are small, thereby 
questioning the validity of their rankings as 
representative on their group as a whole. To overcome 
this weakness, the significance level used was p<.001 
(Miller 1984) . 

Most of the nationality groupings place "Well- 
organised and clearly presented lectures (G) " as number 
1, closely followed by "Explains concepts, principles and 
abstract theories clearly and precisely (H)"; "Is well- 
prepared for classes (D)"; "Good knowledge of subject 
(F)"; and "Structures difficult topics in easily 
understandable ways (E)". A Kruskal-Wallis 1 way ANOVA 
found significant differences on two characteristics: 

i) "States objectives clearly at beginning of course 
(L) " - (X 2 = 27.6; p<.001) rated higher by African 
students (N = 29) . But all significantly higher except 
Arabic-speaking students (N = 6) . 

ii) "Sense of humour (T)" - (X 2 = 26.0; p<.001) 
rated higher by South-east Asian students (N = 53) than 
all others, except Caribbean and North American students 
(N = 5) . Not significant, though, with Arabic-speaking 
and European students (N = 6 and 17 respectively) . 

All the nationality groups, except South-east Asia 
and African, were combined. A Kruskis-Wallis between 
those nationality groups and the rest found one further 
significant differences. "Good general knowledge (C) " 
rated higher by South-East Asian students (X 2 = 14.4; 
p<.001) . Analysis by independent t-test of South-east 
Asia and Africa groups found no other significant 
differences . 

From South-east Asian students, there were 71 
responses; this allowed a sub-division into individual 
nationalities . 

A comparison of Japanese (19 responses) and Hong 
Kongee (17) responses produced no significant 
differences between them, nor between Japanese students 
and the rest of the South-east Asian group, on all 
characteristics. Practically though, Japanese students 
placed "speaks clearly and audibly" (P) higher, and "is 
well-prepared for classes" (D) lower than Hong Kong 
students. The numbers of the other individual countries 
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in this group were too small to use. Even the Japanese 
and Hong Kong samples are relatively small admittedly. 

Factor analysis was performed on all the responses. 
The literature reviewed had found a number of factors. 
The data from the questionnaire was tested by the 
Bartlett test of sphericity, producing a significant 
level of p< . 00001. This suggested the use of the factor 
model (Maria and Norusis 1988) . But the anti-image 
correlation matrix produced a large number of high 
coefficients. This suggested against the use of the 
factor model (Maria and Norusis 1988) . Also, practically 
from looking at the correlation matrix, few correlations 
between characteristics were very large. Less than five 
of the correlations were greater than 0.3 (Maria and 
Norusis 1988) . Not surprisingly, factor analysis did not 
produce any clear-cut results (thirteen different 
approaches were tried) . An example of the factor analysis 
can be seen in Appendix 3 . 

However, the fact that the results of the study at 
the University of Birmingham would not reduce to a 
smaller number of factors is not seen as a problem. The 
use of factor analysis generally, is not without 
difficulties . 

Rees (1990) lists the three main objections to the 
use of factor analysis generally: 

i) "Factor" output "wholly dependent upon the data 
entered into the calculation" (pll) . 

ii) There are many possible factor analytic 
solutions . 

iii) The number of factors to extract, and the 
naming of them is subjective. 

Furthermore, it highlights the fact that the choice 
of characteristics of the ideal lecturer by overseas 
students may be different to previous studies, which 
probably all used home students. This adds support to the 
need for the construction of a separate ratings 
instrument for overseas students. 

The final version of the Birmingham Overseas 
Students Teaching Evaluation Questionnaire (BOSTEQ) can 
be found in Appendix 4 . 
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APPENDIX 1 

ITEMS USED IN PILOT STUDY 

A - Makes good use of handouts. 

B - Deals fairly and impartially with all students. 

C - Discusses recent developments in subject. 

D - Has good general knowledge. 

E - Is well-prepared for classes. 

F - Structures difficult topics in easily understandable ways. 

G - Has good knowledge of own subject. 

H - Encourages class members to work as a team. 

I - Well-organised and clearly presented lectures. 

J - Encourages all class members to contribute to discussions. 

K - Provides full set of references for studies guoted. 

L - Explains concepts, principles, and abstract theories clearly and 

precisely . 
M - Relates course material to other subjects, everyday life or 

future career. 
N - Avoids excess of factual material. 

- Encourages and stimulates students to independent thought. 
P - Includes material not readily available in textbooks. 

Q - Covers course topics in sufficient depth. 

R - Answers guestions accurately. 

S - Clearly states objectives of course at beginning. 

T - Finds ways to help students answer their own guestions. 

U - Clearly defines student responsibility on the course. 

V - Welcomes guestions and discussion from students. 
W - Makes clear how each topic fits in course. 

X - Encourages students to express themselves freely and openly, and 

takes their views into account. 

Y - Always well-groomed, neat and clean in personal appearance. 

Z - Provides adeguate feedback with helpful/constructive comments on 

work . 
Al - Concerned about effectiveness of teaching. 
Bl - Returns written work promptly. 
CI - Maintains continuity in course. 
Dl - Speaks clearly and audibly during lectures. 
El - Readily available to help outside classes. 
Fl - Accomplishes objectives of course, 
Gl - Has good vocabulary. 
HI - Awards grades suitably and fairly based on work performed. 

II - Stresses important material on course. 

Jl - Indicates how much attention to give to different parts of 

reading material. 
Kl - Is aware when students having difficulty in understanding 

topics . 
LI - Speaks with expression, and varies tone of voice during 

lectures . 
Ml - Avoids trivial time-filling material. 
Nl - Writes legibly on blackboard. 

01 - Seems genuinely interested in students. 
PI - Skilful in blackboard drawing. 

Ql - Stresses application as well as theory. 

Rl - Contrasts varying views objectively. 

SI - Stimulates interest in subject beyond course reguirements . 

Tl - Invites criticism of own ideas. 

Ul - Makes good use of examples and illustrations in lectures. 

VI - Motivates students to do best work. 

Wl - Speaks at pace suitable for maximum comprehension, and 
facilitates note-taking. 

XI - Shows interest and enthusiasm in own subject. 

Yl - Tries to link lectures with other parts of course. 
Zl - Has sense of humour. 
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A2 - Keeps strictly to textbook in lectures. 

B2 - Provides all students need to pass examinations. 

C2 - Entertaining to listen to in lectures. 

D2 - Punctual and reliable in attendance at classes. 

E2 - Changes approach to meet new situations. 

F2 - Free from annoying personal peculiarities. 

G2 - Indicates where relevant information not dealt with in the 

lecture can be found. 
H2 - Demands reasonable amount of work. 
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APPENDIX 2 

COPY OF FINAL QUESTIONNAIRE 
PART 1: General Questions. 
What is your nationality? 
What is your course? 

What year of your course are you studying now? 
How long have you been in the UK? 
Is this your first time to study here? 
Are you male or female? 
Which age group are you? 18-21; 22-30; 31-40; 41-50; 51+ 



PART 2: Listed below are 22 characteristics that an ideal 
lecturer could possess. Please rank them in terms of importance, 
starting with 1 as most important. 

A Makes good use of handouts. 

B Discusses recent developments in subject. 

C Has good general knowledge. 

D Is well-prepared for classes. 

E Structures difficult topics in easily understandable 

ways . 

F Has good knowledge of own subject. 

G Well-organised and clearly presented lectures. 

H Explains concepts, principles, and abstract theories 

clearly and precisely. 

I Relates course material to other subjects, everyday life 

or future career. 
J Encourages and stimulates students to independent 

thought . 

K Answers guestions accurately. 

L Clearly states objectives of course at beginning. 

M Encourages students to express themselves freely and 

openly, and takes their views into account. 

N Provides adeguate feedback with helpful/constructive 

comments on work. 

Includes relevant personal anecdotes in lecture. 

P Speaks clearly and audibly during lectures. 

Q Is aware when students having difficulty in understanding 

topics . 

R Motivates students to do best work. 

S Shows interest and enthusiasm in own subject. 

T Has sense of humour. 

U Welcomes guestions and discussion from students. 

V Has good vocabulary. 
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APPENDIX 3 

EXAMPLE OF FACTOR ANALYSIS ON DATA FROM QUESTIONNAIRE AT 
UNIVERSITY OF BIRMINGHAM 

The SPSS default method (principal components with varimax 
rotation) is typical of the other methods used. Factors are 
produced, but it very hard to name them. For example, factor I 
includes "provides adequate feedback with helpful/constructive 
comments (N)", "aware when students having difficulty in 
understanding (Q)", and "welcomes questions and comments from 
students (U) " which logically could be called "relationship with 
students", but also includes a negative "explains concepts, 
principles and abstract theories clearly and precisely (H) " . 



Items I II III IV V VI VII VIII IX 

V -77 

N 68 

Q 59 -31 

S 85 

A -66 

J 82 

M 73 

F -76 

D -30 -62 39 

E -33 55 

I 75 

U -64 

35 60 37 

H -78 

R 68 

B 65 

C -31 -33 -63 

T -51 43 

L -79 

G 57 47 

K -39 -30 55 

P 74 



Rotated factor matrix on SPSS default factor analysis. The 
item letters relate to characteristics in final questionnaire, 
and are described in detail in appendix 2. 
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APPENDIX 4 



BIRMINGHAM OVERSEAS STUDENTS TEACHING EVALUATION QUESTIONNAIRE 

NATIONALITY: 

COURSE: 

SEX: 

LECTURER RATED: 



Using the following key, rate the lecturer on each of the 
statements . 

1 = Poor 4 = Just Above Average 

2 = Below Average 5 = Good 

3 = Average 6 = Excellent 



1. Makes good use of handout. 

2. Is well-prepared for classes. 

3. Structures difficult topics in easily understandable ways. 

4. Has good knowledge of own subject. 

5. Well-organised and clearly presented lectures. 

6. Explains concepts, principles, and abstract theories clearly 

and precisely. 

7. Encourages and stimulates students to independent thought. 

8. Clearly states objectives of course at beginning. 

9. Encourages students to express themselves freely and openly, 

and takes their views into account. 

10. Provides adeguate feedback with helpful/constructive comments 

on work. 

11. Speaks audibly and clearly during lectures. 

12. Is aware when students having difficulty in understanding 

topics . 

13. Overall evaluation of lecture. 

ANY FURTHER COMMENTS ABOUT THE LECTURE. 
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